[Python] How do I read binary pickle data first, then unpickle it?

Posted by conradlee on Stack Overflow See other posts from Stack Overflow or by conradlee
Published on 2010-05-04T09:47:09Z Indexed on 2010/05/04 9:58 UTC
Read the original article Hit count: 256

Filed under:

python

|

serialization

|

pickle

|

unpickling

I'm unpickling a NetworkX object that's about 1GB in size on disk. Although I saved it in the binary format (using protocol 2), it is taking a very long time to unpickle this file---at least half an hour. The system I'm running on has plenty of system memory (128 GB), so that's not the bottleneck.

I've read here that pickling can be sped up by first reading the entire file into memory, and then unpickling it (that particular thread refers to python 3.0, which I'm not using, but the point should still be true in python 2.6).

How do I first read the binary file, and then unpickle it? I have tried:

import cPickle as pickle
f = open("big_networkx_graph.pickle","rb")
bin_data = f.read()
graph_data = pickle.load(bin_data)

But this returns:

TypeError: argument must have 'read' and 'readline' attributes

Any ideas?

© Stack Overflow or respective owner

Related posts about python

unmet dependencies in Ubuntu 12.04

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I tried today to install a dvb-card on my Ubuntu 12.04 (Linux blauhai-linux 3.2.0-25-generic #40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux ). The installation failed with an error. After that, i tried to install python (it was already installed but i got this error): linux:~$… >>> More
How can I get sikuli-ide to work?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I installed sikuli-ide with sudo apt-get install sikuli-ide Everything was fine until I tried to start it from the terminal. I typed sikuli-ide But the only response I got was [info] locale: en_US The application was not started, furthermore there is no desktop file and sikuli-ide does not… >>> More
Getting PATH right for python after MacPorts install

as seen on Super User - Search for 'Super User'
I can't import some python libraries (PIL, psycopg2) that I just installed with MacPorts. I looked through these forums, and tried to adjust my PATH variable in $HOME/.bash_profile in order to fix this but it did not work. I added the location of PIL and psycopg2 to PATH. I know that Terminal is… >>> More
call python with system() in R to run a python script emulating the python console

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to pass a chunk of Python code to Python in R with something like system('python ...'), and I'm wondering if there is an easy way to emulate the python console in this case. For example, suppose the code is "print 'hello world'", how can I get the output like this in R? >>> print… >>> More
Python - Calling a non python program from python?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am currently struggling to call a non python program from a python script. I have a ~1000 files that when passed through this C++ program will generate ~1000 outputs. Each output file must have a distinct name. The command I wish to run is of the form: program_name -input -output -o1 -o2… >>> More

Related posts about serialization

XML Serialization and Soap Serialization

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi I think even if we will not need interoperability between applications, and even we do not communicate with web services, it is easier to serialize using SoapFormatter than XmlSerializer because SOAP will serialize the private members by default, while XmlSerializer will work on public properties… >>> More
Binary serialization/de-serialization in C++ and C#

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello. I am working on a distributed application which has two components. One is written in standard C++ (not managed C++) and the other one is written in C#. Both are communicating via a message bus. I have a situation in which I need to pass objects from C++ to C# application and for this I need… >>> More
Insert some data at later point once serialization(binary serialization) is done for a class

as seen on Stack Overflow - Search for 'Stack Overflow'
My application is written in C# and I am a newbie to C#. I am facing with a problem during serialization. The problem is as follows. There are many classes in my project and when serialization is called on one of the main class object it serializes itself and during this process many of the member… >>> More
FormatException with IsolatedStorageSettings

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a problem when serializing a Dictionary<string,Person> to IsolatedStorageSettings. I'm doing the following: public Dictionary<string, Person> Names = new Dictionary<string, Person>(); if (!IsolatedStorageSettings.ApplicationSettings.Contains("Names")) { … >>> More
Serialization of Queue type- Serialization not working; C#

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi All, Consider this piece of code: private Queue Date=new Queue(); //other declarations public DateTime _Date { get { return (DateTime)Date.Peek();} set { Date.Enqueue(value); } } //other properties and stuff.... public void UpdatePosition(...) { //other code … >>> More